Information processing apparatus, image capturing apparatus, and control methods for the same

ABSTRACT

The information processing apparatus which plays video sent from a image capturing apparatus, comprises a list requesting unit which sends a request for segment list to the image capturing apparatus to acquire the segment list, a determination unit that determines a segment to be requested among the segments indicated in the acquired segment list, a first segment requesting unit that sends a request for the determined segment to the image capturing apparatus, a first segment acquisition unit that acquires the requested segment, and a playing unit that plays the acquired segment, wherein the list requesting unit sends a request for a latest segment list to the image capturing apparatus, whenever the segment is acquired a predetermined number of times by the first segment acquisition unit, and the determination unit determines a segment to be requested, based on the latest segment list.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 14/095,420,filed Dec. 3, 2013, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an information processing apparatus, animage capturing apparatus, and a control method for the same.

Description of the Related Art

In conventional live streaming, since the real-time quality isconsidered important, the RTP (Real-time Transport Protocol)/UDP (UserDatagram Protocol) protocol has been used in the transfer of streamdata. This live streaming is typically utilized for viewing. However,this method can result in data loss and the like, and is not suitable incases such as where it is desired to record stream data in the receivingterminal apparatus. In contrast, there is a method that switchesoperation between a normal mode in which the camera continuouslytransmits captured video and a file transfer mode in which the camera,under instruction from a client, temporarily stores captured video as afile and transfers the file (Japanese Patent No. 4341616).

On the other hand, a method that realizes streaming by utilizing segmentdata obtained by dividing the streaming data into segments of fixedduration and a playlist containing this information is also conceivable.However, in the above conventional example, there is a problem in thatsince HTTP is used as the communication protocol and long delays canoccur depending on network conditions, the playlist method is notsuitable for live viewing due to delays in segment acquisition.

SUMMARY OF THE INVENTION

This specification in its first aspect provides an informationprocessing apparatus that has a communication unit for communicatingwith an image capturing apparatus and plays video data captured by theimage capturing apparatus by sequentially acquiring segments, which areeach a portion of the video data, the apparatus comprising: a listrequesting unit that sends a request for a segment list to the imagecapturing apparatus via the communication unit, where the segment listincludes segment information of video data that is being captured by theimage capturing apparatus; a list acquisition unit that acquires thesegment list requested by the segment list requesting unit; adetermination unit that determines a segment to be requested among thesegments indicated in the segment list acquired by the list acquisitionunit; a first segment requesting unit that sends a request for thesegment determined by the determination unit to the image capturingapparatus via the communication unit; a first segment acquisition unitthat acquires the segment requested by the first segment requestingunit; and a playing unit that plays the segment acquired by the firstsegment acquisition unit; wherein the list requesting unit sends arequest for a latest segment list to the image capturing apparatus,whenever the segment is acquired a predetermined number of times by thefirst segment acquisition unit, and wherein the determination unitdetermines a segment to be requested, based on the latest segment list.

This specification in its second aspect provides an image capturingapparatus that has an image capturing unit and sequentially transmitssegments, which are each a portion of video data obtained by the imagecapturing unit, while performing image capture using the image capturingunit, the apparatus comprising: a management unit that manages segmentsobtained by the image capturing unit, using a segment list; a listtransmission unit that transmits the segment list in response to asegment list request from an information processing apparatus; and asegment transmission unit that transmits a segment requested by theinformation processing apparatus among the segments managed in thesegment list, wherein the management unit adds new segment informationto the segment list, whenever a new segment is obtained by the imagecapturing unit, and wherein the management unit deletes the segmentinformation of the segment transmitted to the information processingapparatus by the segment transmission unit from the segment list.

This specification in its third aspect provides a method for controllingan information processing apparatus that has a communication unitconfigured to communicate with an image capturing apparatus and playsvideo data captured by the image capturing apparatus by sequentiallyacquiring segments, which are each a portion of the video data,comprising: a list requesting step of sending a request for a segmentlist to the image capturing apparatus via the communication unit, wherethe segment list includes segment information of video data that isbeing captured by the image capturing apparatus; a list acquisition stepof acquiring the segment list requested in the segment list requestingstep; a determination step of determining a segment to be requestedamong the segments indicated in the segment list acquired in the listacquisition step; a first segment requesting step of sending a requestfor the segment determined by the determination step to the imagecapturing apparatus via the communication unit; a first segmentacquisition step of acquiring the segment requested in the first segmentrequesting step; and a playing step of playing the segment acquired inthe first segment acquisition step, wherein, in the list requestingstep, a request for a latest segment list is sent to the image capturingapparatus, whenever the segment is acquired a predetermined number oftimes in the first segment acquisition step, and wherein, in thedetermination step, a segment to be requested is determined based on thelatest segment list.

This specification in its fourth aspect provides a method forcontrolling an image capturing apparatus that has an image capturingunit and sequentially transmits segments, which are each a portion ofvideo data obtained by the image capturing unit, while performing imagecapture using the image capturing unit, comprising: a management step ofmanaging segments obtained by the image capturing unit, using a segmentlist; a list transmission step of transmitting the segment list inresponse to a segment list request from an information processingapparatus; and a segment transmission step of transmitting a segmentrequested by the information processing apparatus among the segmentsmanaged in the segment list, wherein, in the management step, newsegment information is added to the segment list, whenever a new segmentis obtained by the image capturing unit, and wherein, in the managementstep, the segment information of the segment transmitted to theinformation processing apparatus in the segment transmission step isdeleted from the segment list.

According to the present invention, in an image capturing apparatuscapable of communicating with an external apparatus, it is possible tosave data without data loss while suppressing any effect on viewing,particularly in the case of recording streaming data to the externalapparatus by live streaming.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an exemplary configuration of a camera 1000to which a first embodiment is applicable.

FIG. 2 is a diagram showing an exemplary configuration of a terminalapparatus 2000 to which the first embodiment is applicable.

FIG. 3 is a diagram showing a communication sequence in a communicationsystem according to the first embodiment.

FIG. 4 is a diagram showing an exemplary playlist according to the firstembodiment.

FIGS. 5A and 5B are diagrams showing an exemplary transition of aplaylist including at a time of congestion according to the firstembodiment.

FIG. 6 is a diagram showing a flowchart of processing by the camera 1000according to the first embodiment.

FIG. 7 is a diagram showing a flowchart of processing by the terminalapparatus 2000 according to the first embodiment.

FIG. 8 is a diagram showing an exemplary screen of the terminalapparatus 2000 according to the first embodiment.

FIGS. 9A and 9B are diagrams showing an exemplary transition of aplaylist including at a time of congestion according to a secondembodiment.

FIG. 10 is a diagram showing a flowchart of processing by the camera1000 according to the second embodiment.

FIG. 11 is a diagram showing a flowchart of processing by the terminalapparatus 2000 according to the second embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments according to the present invention will bedescribed in detail in accordance with the accompanying drawings.

First Embodiment

FIGS. 1 and 2 show exemplary configurations of a camera 1000 serving asan image capturing apparatus and a terminal apparatus 2000 serving as aninformation processing apparatus that displays video received from thecamera 1000, the camera 1000 and the terminal apparatus 2000 beingconstituent members of a video processing system of a first embodiment.As the image capturing apparatus, a camera mobile phone or a so-calledtablet can also be used, besides a still camera or a video camera whosemain function is so-called image capture. Information processingapparatuses including a mobile phone such as a so-called smart phone, aso-called tablet, a TV and the like can also be used for the terminalapparatus 2000, besides a typical personal computer.

In the camera 1000 of FIG. 1, a CPU (Central Processing Unit) 1001, aROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an inputprocessing unit 1004, an output processing unit 1006, a communicationcontroller 1008, a recording medium controller 1011, a camera signalprocessor 1015 and an encoding/decoding processing unit 1016 areconnected to an internal bus 1010. The constituent elements connected tothe internal bus 1010 are configured so as to be able to exchange datawith each other via the internal bus 1010.

The ROM 1002 has stored therein various programs and settings data foroperations of the CPU 1001. A flash memory or the like is also included.In the RAM 1003, programs, variables, temporary data for tasks and thelike that are required during operations of the CPU 1001 are stored asappropriate.

The CPU 1001 controls the constituent elements of this camera 1000 inaccordance with programs stored in the ROM 1002 or a recording medium1012, using the RAM 1003 as a work memory.

An optical system 1013 is an image capturing lens that includes a focusmechanism, a diaphragm mechanism and the like, and forms an opticalimage of a subject. An image capturing device 1014 is constituted by aCCD sensor, a CMOS sensor or the like, and here includes an A/Dconvertor and converts the optical image into an analog electricalsignal, and then converts the analog electrical signal into a digitalsignal.

The camera signal processor 1015 performs processing such as resizingthat involves predetermined pixel interpolation and reduction, colorconversion and various correction processing on the digital signalconverted by the image capturing device 1014, under the control of theCPU 1001.

The encoding/decoding processing unit 1016 compression encodes thedigital signal processed by the camera signal processor 1015 into apredetermined format at a predetermined bit rate, or decodes compressionencoded video data, under the control of the CPU 1001.

Note that although audio is not particularly illustrated, in the case ofdistributing video with audio, a microphone, an A/D convertor fordigitizing audio analog signals and a configuration for encoding digitaldata will be provided. Naturally, at the time of video recording, audiowill be simultaneously recorded together with the video, and video datawith audio will be generated by the encoding/decoding processing unit1016 multiplexing the video and audio.

The input processing unit 1004 receives a user operation on an operationunit 1005, generates a control signal according to the operation, andsupplies the control signal to the CPU 1001. For example, the operationunit 1005 has a text input device such as a keyboard, a pointing devicesuch as mouse or a touch panel, and the like, as input devices forreceiving user operations. Also, a remotely operable device such as aninfrared remote control is also included. Note that a touch panel is aninput device that is configured such that coordinate information thatdepends on the position touched on an input unit having a planerconfiguration, for example, is output. The camera 1000 can thereby beoperated according to a user operation.

The output processing unit 1006 outputs a display signal for causingdisplay to a display unit 1007, based on display data of a GUI(Graphical User Interface) or the like generated by the CPU 1001 inaccordance with a program.

Note that in the case of using a touch panel as the operation unit 1005,the operation unit 1005 and the display unit 1007 can be integrated asone. For example, the touch panel is configured so that thetransmissivity of light is high enough to not interfere with the displayof the display unit 1007, and is attached to an upper layer of thedisplay surface of the display unit 1007. The input coordinates of thetouch panel and display coordinates on the display unit 1007 are alsomapped to each other. A GUI that makes it appear as if the screendisplayed on the display unit 1007 is directly operable by the user canthereby be configured.

The recording medium controller 1011 has connected thereto a recordingmedium 1012 such as an HDD or a nonvolatile semiconductor memory, andperforms reading of data from the connected recording medium 1012 andwriting of data to the recording medium 1012, under the control of theCPU 1001. Note that a recording medium 1012 that is connectable to therecording medium controller 1011 may be a removable nonvolatilesemiconductor memory such as a memory card, for example, that isconnected via a socket or the like not shown.

The recording medium 1012 can also record information required forcontrol by the CPU 1001, besides captured video data.

The communication controller 1008 communicate with an external apparatus(terminal apparatus 2000 in the embodiments), via a connector(wired)/antenna (wireless) 1009, under the control of the CPU 1001.Communication can be performed using standards such as IEEE 802.11 andBluetooth for wireless communication and IEEE 802.3 for wiredcommunication.

Since most of the configuration of the terminal apparatus 2000 in FIG. 2is similar to the camera 1000, the description of constituent elementsfrom a CPU 2001 to a recording medium 2012 that overlaps with the camera1000 will be omitted. An encoding/decoding processing unit 2013 decodescompression encoded video data, and re-encodes decoded data if required,under the control of the CPU 2001.

FIG. 3 is a diagram showing an exemplary communication system accordingto the first embodiment. The communication system of the presentembodiment executes live streaming utilizing a playlist. First, thebasic operations of streaming utilizing a playlist are as follows.

Process 1: A server (camera 1000 in the embodiments) divides streamingdata into segments of fixed duration, and creates a playlist that lists(describes) the acquisition locations of the segments. In the case oflive streaming, since segments are generated periodically, a slidingwindow playlist that is dynamically updated (by deletion/addition) whena new segment is generated into a playlist having new contents is used.

Process 2: A client (terminal apparatus 2000 in the embodiments)acquires and analyzes the playlist, and acquires data from the segmentdata acquisition locations in list order.

Process 3: The client plays/displays or saves the acquired data.

Process 4: The server and the client repeat processes 1 to 3 until theend of the playlist (end of streaming).

Hereinafter, outline operations of live streaming utilizing a playlistwill be described.

On the basis on the basic operations described above, outline operationsof live streaming utilizing a playlist in the present embodiment willnow be described using FIGS. 1, 2, 3 and 8.

When the user executes a live streaming mode from the operation unit1005 in the camera 1000, the camera 1000 sets the communicationcontroller 1008 to a communicable state, under the control of the CPU1001.

Furthermore, the user launches an application required forcommunication/connection processing and live streaming by operating anoperation unit 2005 in the terminal apparatus 2000. In response, the CPU2001 of the terminal apparatus 2000 controls a communication controller2008, starts communication with the camera 1000, and performs connectionprocessing, in accordance with a program stored in a ROM 2002 or arecording medium 2012.

Here, the camera 1000 and the terminal apparatus 2000 use HTTP(Hypertext Transfer Protocol) as a communication protocol. Also, UPnP(Universal Plug and Play) is supported in communication/connection. TheUPnP-compatible terminal apparatus 2000, when connecting to a network,sets an IP (Internet Protocol) address using DHCP (Dynamic HostConfiguration Protocol) or Auto IP. The terminal apparatus 2000, havingacquired an IP address, performs a device search and acquiresinformation on the type, service function and the like of devices thatrespond, using “Device Discovery and Control”, in order to mutuallyrecognize other devices on the network (step 3003). The camera 1000sends back device information, playlist acquisition location informationof device-specific information, and the like in response to a devicesearch request from the terminal apparatus 2000 (step 3004).

When the processing for connecting the camera 1000 and the terminalapparatus 2000 has been completed, the camera 1000 starts livestreaming.

FIG. 8 is an exemplary live streaming application screen on the terminalapparatus 2000, and shows an example in which the operation unit 2005and a display unit 2007 are integrated as one. Live streaming data isdisplayed in a display area 8001, and the camera status is indicated byzoom position information (8002), recording state (8003), batteryinformation (8004), and the like.

In live streaming, the CPU 1001 of the camera 1000 starts signal outputfrom the image capturing device 1014, processes the output into suitablevideo data using the camera signal processor 1015, and passes the datato the encoding/decoding processing unit 1016. At this time, the CPU1001 also passes information relating to the camera status such as zoommagnification (or focal length) to the encoding/decoding processing unit1016.

The encoding/decoding processing unit 1016 compression encodes thereceived video data, camera status information and the like intopredetermined format at a predetermined bit rate, and, furthermore,divides the video data into segments of a predetermined duration Ts, andsaves the resultant data as segment data (3002) in the RAM 1003 or therecording medium 1012. Note that, in the present embodiment, Ts=0.5 sec.Accordingly, to be precise, in the terminal apparatus 2000, video isreceived and played or recorded Ts seconds (0.5 seconds) after beingcaptured by the camera 1000.

The CPU 1001 generates path information associated with the segment datastorage location. The path information is used as acquisition locationinformation when the terminal apparatus 2000 performs segmentacquisition, and the CPU 1001 creates a playlist 3001 and records thesegment information to the playlist together with the path information.

Here, the playlist 3001 will be described in detail. FIG. 4 is anexemplary playlist according to the first embodiment.

A playlist 4010 is in Extended M3U format, with an identifier tag beingdescribed in the first line (4011), and a tag showing the playlistversion and the version being described in the second line (4012). Theversion is “3” in present example. A tag showing the duration of segmentdata 3002 and the duration (sec) in integer or decimal form aredescribed in the third line (4013). In the present embodiment, giventhat segment data duration is set as Ts=0.5 (sec), the segment dataduration is also given as “0.5” in the present example. The acquisitionlocation path (includes query parameter) of the segment data 3002 isdescribed in the fourth line (4014). The third line (4013) and thefourth line (4014) are typically required to be described consecutivelyas they contain information relating to the segment data 3002.

A playlist 4010 is exemplary contents of the playlist 3001 of FIG. 3 inwhich segment information (4013, 4014) is recorded.

The terminal apparatus 2000 is assumed to store the predeterminedduration Ts of the segment data 3002 in advance, or be able to obtainthe predetermined duration Ts at the time of device informationacquisition at step S3004 from the device information of the camera1000.

The terminal apparatus 2000 sends a playlist acquisition request (HTTPGET method) to the playlist acquisition location acquired at step 3004after approximately Ts (sec) (step 3005) after the start of livestreaming.

The camera 1000 transmits the playlist 3001 (=4010) in which one segmentinformation set (4013, 4014) is described to the terminal apparatus 2000as a response playlist (step S3006).

The terminal apparatus 2000 analyzes the received playlist 4010 andsends a segment acquisition request (HTTP GET method) to the camera 1000which is the acquisition location of segment information (step 3007).

The camera 1000 transmits the requested segment 3002 as a responsesegment (step 3008).

The terminal apparatus 2000, after the received segment 3002 has beenpassed to the encoding/decoding processing unit 2013 and decoded,performs play/display on the display unit 2007 via the output processingunit 2006. When there is input of an instruction from the user to aterminal apparatus REC 8007, a data portion obtained by excluding theheader and the like from the decoded data or the segment 3002 isrecorded and saved to the recording medium 2012 by the live streamingapplication. The sequentially received segment data sets are then mergedand recorded.

During streaming, the camera 1000 generates segments and updates theplaylist approximately every Ts (sec). Also, the camera 1000 deletessegment information acquired by the terminal apparatus 2000 from theplaylist.

The terminal apparatus 2000, whenever the playlist is updated by thecamera 1000, acquires the updated playlist (step 3005) approximatelyevery Ts (sec), and requests acquisition of corresponding segment data,based on the segment information contained in the playlist 3001.

When communication conditions are good, the series of processing fromplaylist acquisition (step 3005) to the response segment (step 3008) isperformed periodically. However, in actuality, it periodically becomesimpossible to perform communication due to congestion and the like.FIGS. 5A and 5B show the transition of a playlist when congestionoccurs. Reference numerals 5010 to 5050 in the diagram show an exemplaryplaylist at various stages.

Note that an ID unique to the terminal apparatus or application is addedin request processing (steps 3005 to 3007) of the terminal apparatus2000. The camera 1000 of the present embodiment then performs streamingonly in response to a request including the ID attached to the initialrequest. In other words, the camera 1000 and terminal apparatus 2000 ofthe present embodiment only perform streaming in a one-to-oneconnection.

In FIGS. 6 and 7, examples of processing flows applicable to the firstembodiment of the present invention are shown, and description will nowbe given in accordance with these flows. FIG. 6 is a processing flowexecuted by the CPU 1001 after the camera 1000 has established aconnection with the terminal apparatus 2000.

At step S6001, processing for starting generation of segment datafollowing the start of live streaming is performed. The CPU 1001 startssignal output from the image capturing device 1014, processes the outputinto suitable video data using the camera signal processor 1015, andpasses the data to the encoding/decoding processing unit 1016. Theencoding/decoding processing unit 1016 compression encodes the receivedvideo data into a predetermined format at a predetermined bit rate, and,furthermore, starts processing for dividing the video data into segmentsof the predetermined duration Ts (0.5 seconds in the presentembodiment).

At step S6002, it is determined whether generation of one segment (0.5seconds) has been completed. It is checked whether data divided into a0.5 second segment has been temporarily saved in the RAM 1003 or therecording medium 1012. If completed, the processing advances to stepS6003, and if not completed, the processing advances to step S6005. Notethat although this step is described in terms of the determination beingperformed by query (polling), it is also possible to determine thatgeneration has been completed by a method such as notification byinterrupt after detecting that divided segment data has been written.

At step S6003, generation of segment information is performed. Pathinformation associated with the duration of the generated segment dataand a storage location address, filename or the like that is able tospecify the segment data is generated. Here, in the description of thepresent embodiment, the duration of a segment is fixed at 0.5 seconds.

At step S6004, the segment information generated at step S6003 is addedto the playlist. For example, an exemplary playlist 5011 in FIG. 5Abecomes an exemplary playlist 5020, and an exemplary playlist 5030becomes an exemplary playlist 5040.

At step S6005, it is determined whether there is a playlist acquisitionrequest from the terminal apparatus 2000 (step 3005). If there is aplaylist acquisition request, the processing advances to step S6006, andif not, the processing advances to step S6007.

At step S6006 (first transmission processing), in response to theplaylist acquisition request (step 3005), transmission of the playlistis performed (step 3006).

At step S6007, it is determined whether there is a segment acquisitionrequest from the terminal apparatus 2000 (step 3007). If there is asegment acquisition request, the processing advances to step S6008, andif not, the processing returns to step S6002.

At step S6008, it is determined whether the segment requested at stepS6007 is valid. If the segment is valid, the processing advances to stepS6009, and if invalid, the processing advances to step S6010.

At step S6009 (second transmission processing), in response to thesegment acquisition request (step 3007), transmission of thecorresponding segment (step 3008) is performed. Thereafter, the segmentthat has been transmitted is deleted, information relating to thesegment is deleted from the playlist, and the processing returns to stepS6002. For example, an exemplary playlist 5010 becomes an exemplaryplaylist 5011, and an exemplary playlist 5040 becomes an exemplaryplaylist 5050.

At step S6010, in response to the segment acquisition request (step3007), error status transmission (step 3008) is performed, and theprocessing returns to step S6002.

Note that although the event determination steps S6002, S6005 and S6007in the present flow are described in terms of the determination beingperformed by query in order, the individual processing may be executedin order of event occurrence in the case of simultaneously performingevent queuing for various conditions. As is evident from the abovedescription, it should be noted that the image capturing apparatus 1000performs processing for updating the playlist (management processing)both at the timing at which new segments are acquired from the imagecapturing device 1014 and at the timing at which segments aretransmitted to the terminal apparatus 2000.

FIG. 7 is a processing flow executed by the CPU 2001 after the terminalapparatus 2000 establishes connection with the camera 1000.

At step S7001, playlist acquisition location information is acquiredfrom the device/playlist acquisition location information acquired atthe time of connection (step 3004), and held. At step S7002, it isdetermined whether a predetermined duration has elapsed. If thepredetermined duration has elapsed, the processing advances to stepS7003, and if not, step S7002 is repeated again. Here, the predeterminedduration desirably is a value equivalent to the predetermined durationTs that is generated by the camera 1000.

At step S7003, a playlist acquisition request (step 3005) is sent usingthe playlist acquisition location information acquired at step S7001,and a playlist (P1) acquired from the camera 1000 is analyzed. Analysisof the playlist involves acquiring segment information, after checkingthe playlist format and version using identification tags.

At step S7004, it is determined whether segment information was acquiredat step S7003. If segment information was acquired, the processingadvances to step S7005, and if not, the processing returns to stepS7003.

At step S7005 (first video acquisition processing), if one segmentinformation set was acquired, that segment is taken as the latestsegment. Also, if a plurality of segment information sets were acquired,the segment information set described last in the playlist is taken asthe latest segment. A segment acquisition request (step 3007) is thensent to the acquisition location path of the latest segment information,and the segment is acquired from the camera 1000. It should be notedthat because the latest segment is acquired, segments that are not thelatest segment are in a non-acquirable state at this stage. Processingfor acquiring a segment that has not been acquired originating from thisacquisition of the latest segment is performed at step S7009 discussedlater. Note that in the case where an instruction to save streaming hasalready been set by the user in the terminal apparatus 2000, the CPU2001, at step S7005, performs processing for recording the receivedsegment to the recording medium 2012.

The acquired segment, after being recorded to the RAM 2003 or therecording medium 2012, is passed to the encoding/decoding processingunit 2013 and decoded, after which play/display (video display) isperformed by the display unit 2007 via the output processing unit 2006.Also, in the case where the terminal apparatus REC 8007 is beingexecuted by the live streaming application, a data portion obtained byexcluding the header and the like from the decoded data or the segmentis saved to the recording medium 2012 by separate processing from thepresent processing flow.

At step S7006, it is determined whether an instruction to save videothat is being streamed has been given in the terminal apparatus 2000. Inother words, it is determined whether the terminal apparatus REC 8007 bythe live streaming application that operates on the terminal apparatus2000 is being executed. If a streaming save instruction has been set,the processing advances to step S7007, and if the streamed video is notbeing saved, the processing returns to step S7002. In other words,acquisition of one playlist and acquisition of one segment can be seenas one set of processing for play processing.

At step S7007, it is determined whether a plurality of segmentinformation sets were acquired at step S7003. If a plurality ofinformation sets were acquired, the processing advances to step S7008,and if one information set was acquired, the processing returns to stepS7002.

At step S7008, it is determined whether a predetermined duration haselapsed, similarly to step S7002. If a predetermined duration haselapsed, the processing returns to step S7003, and if not, theprocessing advances to step S7009. Here, the predetermined durationdesirably is set to a value equivalent to the predetermined duration Tsthat is generated by the camera 1000.

At step S7009 (second video acquisition processing), the CPU 2001 sendsa segment acquisition request (step 3007) to the acquisition locationpath of the oldest segment information in the playlist, and acquires thecorresponding segment from the camera 1000.

At step S7009, the acquired segment is recorded to the RAM 2003 or therecording medium 2012. The segment acquired here, being older than thesegment acquired at step S7005, is not used for play, and a data portionobtained by excluding the header and the like from the decoded data orthe segment is saved to the recording medium 2012 by separate processingfrom this processing flow. At this time, processing for merging receivedsegment data sets in the original order, that is, in the temporal orderin which the data was captured, is performed. Seamless data restorationis thereby possible separately from play.

Although a detailed description has been given above based on favorableembodiments according to the present invention, the present invention isnot limited to these specific embodiments, and various configurationswithin a range that does not depart from the gist of this invention arealso included in the present invention. Some of the abovementionedembodiments may also be combined as appropriate.

In the above embodiments, the timing for executing step S7009 is givenas when it is determined that the streamed video is being saved (Yes inS7006), there are a plurality of segments in the playlist (Yes inS7007), and the predetermined duration has not elapsed (No in S7008),but the present invention is not limited thereto. For example, thisprocessing may be performed in the case where an instruction to “endsaving (recording)” is received from the user. As a result, becausesegments captured after the instruction to end saving need not bereceived, segments that have not yet been received can be receivedcollectively, and the segments, when all together, can be saved as asingle file of streaming data.

An example was described in which one of the conditions for executingstep S7009 of FIG. 7 was the requirement that streamed video is beingsaved in the terminal apparatus 2000 (step S7006 is Yes). However, inorder to also enable so-called chasing play in which time shift play isperformed during play, the condition of step S7009 may be omitted.

In the flowchart of FIG. 7, after the latest segment is acquired atS7005, the processing returns to S7003 and the playlist is acquired, buteven in the case where acquisition of the latest segment fails at S7005,the processing returns to S7003, and the playlist is requested andacquired. In other words, the latest playlist is requested whenever thelatest segment is requested.

Second Embodiment

According to the first embodiment, information showing segments that theterminal apparatus 2000 was not able to acquire due to factors such asnetwork congestion are added to the playlist by the camera 1000. Inother words, in some cases, the size of the playlist will grow.

In a second embodiment, an example is described in which an increase inthe size of the playlist is suppressed, by rewriting segments that aretemporally continuous, among the segment that are yet to be transmitted,as a single segment in the camera 1000. Note that the camera 1000 andthe terminal apparatus 2000 are assumed to have similar configurationsto the first embodiment, and description thereof will be omitted. Thepoint that the length Ts of the segments is 0.5 seconds is also the sameas the first embodiment.

FIGS. 9A and 9B show the transition of a playlist at the time of typicalcongestion, for the purposes of illustrating the second embodiment. Thereference numerals 9010 to 9080 show a playlist at various stages.

Note that the camera 1000 and the terminal apparatus 2000 of the presentembodiment also only perform streaming in a one-to-one connection.

Because the flowcharts of FIGS. 10 and 11 replace FIGS. 6 and 7 of thefirst embodiment, the same reference numerals are given to processingthat is the same.

First, the processing flow that is executed by the CPU 1001 after thecamera 1000 has established a connection with the terminal apparatus2000 will be described, in accordance with the flowchart in FIG. 10.

At step S6001, processing for starting generation of segment datafollowing the start of live streaming is performed. The CPU 1001 startssignal output from the image capturing device 1014, processes the outputinto suitable video data using the camera signal processor 1015, andpasses the data to the encoding/decoding processing unit 1016. Theencoding/decoding processing unit 1016 compression encodes the receivedvideo data into a predetermined format at a predetermined bit rate, and,furthermore, starts processing for dividing the video data into segmentsof the predetermined duration Ts (also 0.5 seconds in the secondembodiment). In short, the CPU 1001 performs processing for encoding thevideo data obtained from the image capturing device 1041 in units of 0.5seconds, and storing the encoded data in the RAM 1003 or the recordingmedium 1012 as a single file.

At step S6002, it is determined whether generation of one segment (0.5seconds) has been completed. It is checked whether data divided into a0.5 second segment is saved in the RAM 1003 or the recording medium1012. If completed, the processing advances to step S6003, and if notcompleted, the processing advances to step S6005. Note that althoughthis step is described in terms of the determination being performed byquery (polling), it is also possible to determine that generation hasbeen completed by a method such as notification by interrupt afterdetecting that divided segment data has been written.

At step S6003, segment information is generated. Path informationassociated with the duration of the generated segment data and a storagelocation address, filename or the like that is able to specify thesegment data is generated. Here, in the description of this secondembodiment, the duration of a segment is fixed at 0.5 seconds.

At step S6100, it is determined whether there are two or more segmentsthat the terminal apparatus 2000 has not yet acquired. In other words,it is determined whether two or more segment information sets (4013,4014) are included in the playlist. If two or more segment informationsets (4013, 4014) are included (exemplary playlist 9030), the processingadvances to step S6101, and if two or more segment information sets arenot included (exemplary playlist 9011), or if only one segmentinformation set is included (exemplary playlist 9020), the processingadvances to step S6004.

At step S6101, continuous segments that have not yet been acquired aremerged, and segment duration (segment information) and segment pathinformation are merged. At step S6102, update processing for reflectingthe information merged at step S6101 in the playlist is performed. Notethat the merging of the segments themselves may be performed after asegment request is actually received.

At step S6004, the segment information generated at step S6003 is addedto the playlist. After the end of this step, the segment information ofeach playlist determined at step S6100 will be the latest segmentinformation and, in the case where there are unacquired segments, aplurality of unacquired segment information sets in which continuoussegments are merged. In other words, the exemplary playlist 9030 becomesan exemplary playlist 9040, the exemplary playlist 9011 becomes anexemplary playlist 9020, the exemplary playlist 9020 becomes theexemplary playlist 9030, and an exemplary playlist 9060 becomes anexemplary playlist 9070.

At step S6005, it is determined whether there is a playlist acquisitionrequest (step 3005) from the terminal apparatus 2000. If there is aplaylist acquisition request, the processing advances to step S6006, andif not, the processing advances to step S6007.

At step S6006, in response to the playlist acquisition request (step3005), transmission of the playlist is performed (step 3006).

At step S6007, it is determined whether there is a segment acquisitionrequest (step 3007) from the terminal apparatus 2000. If there is asegment acquisition request, the processing advances to step S6008, andif not, the processing returns to step S6002.

At step S6008, it is determined whether the segment requested at stepS6007 is valid. If the segment is valid, the processing advances to stepS6009, and if invalid, the processing advances to step S6010.

At step S6009, in response to the segment acquisition request (step3007), transmission of the corresponding segment (step 3008) isperformed, after which the segment is deleted, related segmentinformation is deleted from the playlist, and the processing returns tostep S6002. On the other hand, at step S6010, in response to the segmentacquisition request (step 3007), error status transmission (step 3008)is performed, and the processing returns to step S6002.

Note that although the event determination steps S6002, S6005 and S6007in the present flow are described in terms of the determination beingperformed by query in order, the individual processing may be executedin order of event occurrence in the case of simultaneously performingevent queuing for various conditions.

FIG. 11 is a processing flow that is executed by the CPU 2001 after theterminal apparatus 2000 has established a connection with the camera1000.

At step S7001, playlist acquisition location information is acquiredfrom the device/playlist acquisition location information acquired atthe time of connection (step 3004), and held.

At step S7002, it is determined whether a predetermined duration haselapsed. If the predetermined duration has elapsed, the processingadvances to step S7003, and if not, step S7002 is repeated again. Here,the predetermined duration desirably is a value equivalent to thepredetermined duration Ts that is generated by the camera 1000.

At step S7003, a playlist acquisition request (step 3005) is sent, usingthe playlist acquisition location information acquired at step S7001,and a playlist (P1) acquired from the camera 1000 is analyzed. Analysisof the playlist involves acquiring segment information after checkingthe playlist format and version using identification tags.

At step S7004, it is determined whether segment information was acquiredat step S7003. If segment information was acquired, the processingadvances to step S7005, and if not, the processing returns to stepS7003.

At step S7005, if one segment information set was acquired, that segmentis taken as the latest segment. Also, if a plurality of segmentinformation sets were acquired, the last segment information set istaken as the latest segment. A segment acquisition request (step 3007)is then sent to the acquisition location path of the latest segmentinformation, and the segment is acquired from the camera 1000.

The acquired segment, after being recorded to the RAM 2003 or therecording medium 2012, is passed to the encoding/decoding processingunit 2013 and decoded, after which play/display (video display) isperformed by the display unit 2007 via the output processing unit 2006.Also, in the case where the terminal apparatus REC 8007 is beingexecuted by the live streaming application, a data portion obtained byexcluding the header and the like from the decoded data or the segmentis saved to the recording medium 2012 by separate processing from thepresent processing flow.

At step S7006, it is determined whether video that is being streamed isbeing saved in the terminal apparatus 2000. In other words, it isdetermined whether the terminal apparatus REC 8007 by the live streamingapplication that operates on the terminal apparatus 2000 is beingexecuted. If the streamed video is being saved, the processing advancesto step S7100, and if the streamed video is not being saved, theprocessing returns to step S7002.

At step S7100, it is determined whether two or more segment informationsets were acquired at step S7003. If there are two or more informationsets, the processing advances to step S7101, and if there is oneinformation set, the processing returns to step S7002. Here, the numberof segment information sets is given as Sn. At step S7101, similarly tostep S7003, a playlist acquisition request (step 3005) is sent again, aplaylist (P2) is acquired from the camera 1000, and segment informationis acquired.

At step S7102, the number of segment information sets acquired at stepS7101 that have been newly added is determined. If the number ofinformation sets is greater than Sn−1, this means that new segments weregenerated during the period from when the playlist was acquired at stepS7003 until when the playlist was acquired at step S7101, and thus theprocessing returns to step S7005. If the number of information sets isSn−1, it is judged that there are old segments that have not yet beenacquired, and the processing advances to step S7103.

At step S7103, a segment acquisition request is sent to the acquisitionlocation path of unacquired segment information (step 3007), and an oldsegment is acquired from the camera 1000 and recorded to the RAM 2003 orthe recording medium 2012. The segment acquired here, being older thanthe segment acquired at step S7005, is not used for play, and a dataportion obtained by excluding the header and the like from the decodeddata or the segment is saved to the recording medium 2012 by separateprocessing from this processing flow. At this time, merging receivedsegment data sets in the original order enables seamless datarestoration separately from play.

Note that while a configuration in which segments that have not yet beenacquired are collected together as one segment information set wasdescribed in the present embodiment, it is possible with thisconfiguration to increase the size of the segments relative to onesegment information set. In view of this, a configuration may be adoptedin which a threshold, being as upper limit for one segment informationset, such as 5 seconds worth of segments or ten segments per informationset, for example, is set in advance, and if this threshold is exceeded,a new segment information set is created. Subsequent segment informationcan then be merged with the newly created segment information.

In the present embodiment, if a playlist already has two or more segmentinformation sets, the segment information sets are merged.Alternatively, segment information sets may be merged in the case where,for example, there are 2 seconds or more worth of temporally continuoussegments or where there are four or more continuous segments. It shouldbe noted that the values given here, such as 2 seconds and 4 segments,are by way of example.

Although the second embodiment has been discussed in detail above, thepresent invention is not limited to these specific embodiments, andvarious configurations within a range that does not depart from the gistof this invention are also included in the present invention. Some ofthe abovementioned embodiments may also be combined as appropriate.

The case where a software program that realizes the functions of thefirst or second embodiments is supplied to a system or an apparatushaving a computer capable of executing a program directly from arecording medium or using wired/wireless communication and this programis executed is also included in the present invention.

Accordingly, in order to realize the functional processing of thepresent invention with a computer, the present invention is realized bythe program code itself, which is supplied to the computer and installedthereon. In other words, the computer program for realizing thefunctional processing of the present invention is itself included in thepresent invention.

In this case, as long as the functions of a program are realized, anyconfiguration of a program is possible, such as a program that isexecuted by an interpreter, script data that is supplied to an OS, orthe like.

A recording medium for supplying the program may be, for example, a harddisk, a magnetic recording medium such as magnetic tape, anoptical/magneto-optical storage medium, or a nonvolatile semiconductormemory.

Conceivable methods of supplying the program include a computer programfor forming the present invention being stored in a server on a computernetwork, and a client computer connected to the computer networkdownloading and executing the computer program.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2012-269785, filed Dec. 10, 2012, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus that has acommunication unit for communicating with an image capturing apparatusand plays video data captured by the image capturing apparatus bysequentially acquiring segments, which are each a portion of the videodata, said apparatus comprising: a list requesting unit that sends, tothe image capturing apparatus via the communication unit, a request fora segment list, where the segment list includes segment informationindicating at least one segment which the image capturing apparatus hasnot yet sent to the information processing apparatus among segments ofvideo data that is being captured by the image capturing apparatus; alist acquisition unit that acquires, from the image capturing apparatus,the t list requested by the segment list requesting unit; a firstdetermination unit that determines a predetermined segment, as a liveimage to be played, among the segments indicated in the segment listacquired by the list acquisition unit; a first segment requesting unitthat sends a request for the segment determined by the firstdetermination unit to the image capturing apparatus via thecommunication unit; a first segment acquisition unit that acquires thesegment requested by the first segment requesting unit; and a playingunit that plays, as the live image, the segment acquired by the firstsegment acquisition unit.
 2. The information processing apparatusaccording to claim 1, wherein the list requesting unit sends a requestfor a latest segment list to the image capturing apparatus, whenever thesegment is requested or acquired one time by the first segmentrequesting unit or the first segment acquisition unit.
 3. Theinformation processing apparatus according to claim 1, wherein the firstsegment acquisition unit stores the acquired segment into apredetermined storage, wherein the apparatus further comprising: asecond determination unit that determines whether the segment listacquired by the first segment acquisition unit includes informationindicating a past segment other than the latest segment to be played bythe playing unit; and a control unit that, if it is determined that thepast segment is indicated in the segment list acquired by the firstsegment acquisition unit, acquires the past segment by sending a requestfor the past segment to the image capturing apparatus, and stores theacquired past segment into the predetermined storage without playing thepast segment by the playing unit, wherein the control unit merges, intemporal order, the segments stored in the predetermined storage togenerate a seamless video data.
 4. The information processing apparatusaccording to claim 3, wherein the control unit includes: a saving unitthat saves the merged video data.
 5. The information processingapparatus according to claim 4, further comprising a setting unit thatsets whether video data captured by the image capturing apparatus is tobe saved by the saving unit, wherein if saving of video data is set bythe setting unit, the control unit acquires the past segment.
 6. Theinformation processing apparatus according to claim 3, wherein thecontrol unit sends a request for an oldest segment, as the past segment,among the segments indicated in the latest segment list.
 7. Theinformation processing apparatus according to claim 1, wherein thesegment information includes information for specifying a segment andinformation of a time length of the segment.
 8. The informationprocessing apparatus according to claim 1, wherein the informationprocessing apparatus is a mobile phone.
 9. The information processingapparatus according to claim 1, wherein the information processingapparatus is a tablet device.
 10. The information processing apparatusaccording to claim 1, further comprising a setting unit that setswhether video data captured by the image capturing apparatus is to besaved by the saving unit, wherein if saving of video data is set by thesetting unit, the control unit acquires the past segment other than alatest segment, among the segments indicated in the segment list. 11.The information processing apparatus according to claim 3, wherein thecontrol unit sends a request for an oldest segment, as the past segment,among the segments indicated in the segment list.
 12. A method forcontrolling an information processing apparatus that has a communicationunit for communicating with an image capturing apparatus and plays videodata captured by the image capturing apparatus by sequentially acquiringsegments, which are each a portion of the video data, the methodcomprising: a list requesting step of sending, to the image capturingapparatus via the communication unit, a request for a segment list,where the segment list includes segment information indicating at leastone segment which the image capturing apparatus has not yet sent to theinformation processing apparatus among segments of video data that isbeing captured by the image capturing apparatus; a list acquisition stepof acquiring, from the image capturing apparatus, the segment listrequested in the list requesting step; a first determination step ofdetermining a predetermined segment, as a live image to be played, amongthe segments indicated in the segment list acquired in the listacquisition step; a first segment requesting step of sending a requestfor the segment determined in the first determination step to the imagecapturing apparatus via the communication unit; a first segmentacquisition step of acquiring the segment requested in the first segmentrequesting step; and a playing step of playing, as the live image, thesegment acquired in the first segment acquisition step.
 13. Anon-transitory computer-readable storage medium storing a program which,when executed on a computer, causes that computer to perform a methodfor controlling an information processing apparatus that has acommunication unit for communicating with an image capturing apparatusand plays video data captured by the image capturing apparatus bysequentially acquiring segments, which are each a portion of the videodata, the method comprising: a list requesting step of sending, to theimage capturing apparatus via the communication unit, a request for asegment list, where the segment list includes segment informationindicating at least one segment which the image capturing apparatus hasnot yet sent to the information processing apparatus among segments ofvideo data that is being captured by the image capturing apparatus; alist acquisition step of acquiring, from the image capturing apparatus,the segment list requested in the list requesting step; a firstdetermination step of determining a predetermined segment, as a liveimage to be played, among the segments indicated in the segment listacquired in the list acquisition step; a first segment requesting stepof sending a request for the segment determined in the firstdetermination step to the image capturing apparatus via thecommunication unit; a first segment acquisition step of acquiring thesegment requested in the first segment requesting step; and a playingstep of playing, as the live image, the segment acquired in the firstsegment acquisition step.
 14. An information processing apparatus thatcommunicates with an image capturing apparatus and plays video datacaptured by the image capturing apparatus by sequentially acquiringsegments, which are each a portion of the video data, said apparatuscomprising: a processor; a communication interface; wherein theprocessor is configured to function as following units, a listacquisition unit, a first determination unit, a first segment requestingunit, a first segment acquisition unit, and a playing unit, wherein thelist requesting unit that sends, to the image capturing apparatus viathe communication interface, a request for a segment list, where thesegment list includes segment information indicating at least onesegment which the image capturing apparatus has not yet sent to theinformation processing apparatus among segments of video data that isbeing captured by the image capturing apparatus; the list acquisitionunit that acquires, from the image capturing apparatus via thecommunication interface, the segment list requested by the segment listrequesting unit; the first determination unit that determines apredetermined segment, as a live image to be played, among the segmentsindicated in the segment list acquired by the list acquisition unit; thefirst segment requesting unit that sends a request for the segmentdetermined by the first determination unit to the image capturingapparatus via the communication interface; the first segment acquisitionunit that acquires the segment requested by the first segment requestingunit via the communication interface; and the playing unit that plays,as the live image, the segment acquired by the first segment acquisitionunit.
 15. The information processing apparatus according to claim 14,wherein the list requesting unit sends a request for a latest segmentlist to the image capturing apparatus, whenever the segment is requestedor acquired one time by the first segment requesting unit or the firstsegment acquisition unit.
 16. The information processing apparatusaccording to claim 14, wherein the first segment acquisition unit storesthe acquired segment into a predetermined storage, wherein the apparatusfurther comprising: a second determination unit that determines whetherthe segment list acquired by the first segment acquisition unit includesinformation indicating a past segment other than the latest segment tobe played by the playing unit; and a control unit that, if it isdetermined that the past segment is indicated in the segment listacquired by the first segment acquisition unit, acquires the pastsegment by sending a request for the past segment to the image capturingapparatus, and stores the acquired past segment into the predeterminedstorage without playing the past segment by the playing unit, whereinthe control unit merges, in temporal order, the segments stored in thepredetermined storage to generate a seamless video data.
 17. Theinformation processing apparatus according to claim 16, wherein thecontrol unit includes: a saving unit that saves the merged video data.18. The information processing apparatus according to claim 17, furthercomprising a setting unit that sets whether video data captured by theimage capturing apparatus is to be saved by the saving unit, wherein ifsaving of video data is set by the setting unit, the control unitacquires the past segment.
 19. The information processing apparatusaccording to claim 16, wherein the control unit sends a request for anoldest segment, as the past segment, among the segments indicated in thelatest segment list.
 20. The information processing apparatus accordingto claim 14, wherein the segment information includes information forspecifying a segment and information of a time length of the segment.21. The information processing apparatus according to claim 14, whereinthe information processing apparatus is a mobile phone.
 22. Theinformation processing apparatus according to claim 14, wherein theinformation processing apparatus is a tablet device.
 23. The informationprocessing apparatus according to claim 14, further comprising a settingunit that sets whether video data captured by the image capturingapparatus is to be saved by the saving unit, wherein if saving of videodata is set by the setting unit, the control unit acquires the pastsegment other than a latest segment, among the segments indicated in thesegment list.
 24. The information processing apparatus according toclaim 16, wherein the control unit sends a request for an oldestsegment, as the past segment, among the segments indicated in thesegment list.